Imaging and display system for guiding medical interventions

ABSTRACT

An imaging and display system for guiding medical interventions includes a wearable display, such as a goggle display, for viewing by a user. The display presents a composite, or combined image that includes pre-operative surgical navigation images, intraoperative images, and in-vivo microscopy images or sensing data. The pre-operative images are acquired from scanners, such as MRI and CT scanners, while the intra-operative images are acquired in real-time from a camera system carried by the goggle display for imaging the patient being treated so as to acquire intraoperative images, such as fluorescence images. A probe, such as a microscopy probe or a sensing probe, is used to acquire in-vivo imaging/sensing data from the patient. Additionally, the intra-operative and in-vivo images are acquired using tracking and registration techniques to align them with the pre-operative image and the patient to form a composite image for display by the goggle display.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional Application No.61/942,666 filed on Feb. 21, 2014, the content of which is incorporatedherein by reference.

TECHNICAL FIELD

The present invention generally relates to imaging and display systems.Particularly, the present invention relates to an imaging and displaysystem for guiding medical interventions, such as surgicalinterventions. More particularly, the present invention relates to awearable imaging and display system for guiding medical interventions bythe simultaneous display of pre-operative surgical navigation images,real-time intra-operative images, and in-vivo, microscopy imaging andsensing data.

BACKGROUND OF THE INVENTION

Medical professionals, such as surgeons, face enormous challenges duringsurgical interventions. To assist surgeons in their efforts to provideefficient and effective surgical care, three independent and separateapproaches for guiding medical or surgical interventions, or providingmedical or surgical guidance, are currently used.

The first approach for providing surgical guidance is typically referredto as “conventional” surgical navigation, and involves the use ofpre-operative images of a target of interest (TOI), such as brain tumorimages for example, which are captured before surgery takes place. Inaddition to the use of pre-operative images, surgical navigation tracksthe position of surgical instruments relative to these pre-operativeimages, allowing the surgeon to view the movement of the surgicalinstruments relative to the pre-operative images. In other words,surgical navigation provides a visual display for the surgeon, whereuponthe location and movement of the surgical tools relative to thepre-operative images is displayed for the benefit of the surgeon. Thepre-operative images may include various image types, including X-raycomputed tomography (CT) images or magnetic resonance imaging (MRI) forexample.

The second approach of providing surgical guidance includes the use ofintra-operative images, which are images that are acquired, inreal-time, while a surgical procedure is being performed on a patient.For example, fluoroscopy and ultrasound are two well-knownintraoperative imaging technologies that are used for providingintra-operative based surgical navigation. There has also been a desirefrom the surgical community for the use of optical imaging whenproviding intra-operative surgical guidance.

The third approach of surgical guidance is based on using amicroscopy/pathology report. For example, in the case of tumorresection, the surgeon will selectively remove a tissue specimen from atarget tissue and send it to a pathologist for analysis. Duringanalysis, the tissue is sectioned, stained and examined under amicroscope, whereupon the pathologist advises the surgeon as to whetherthere are any residual cancerous cells in the tissue.

However, conventional surgical navigation, intra-operative imaging-basedmedical guidance, and pathology-based medical guidance techniques have avariety of drawbacks. For example, because conventional surgicalnavigation is based on pre-operative images, it is therefore unable toaccommodate the tissue deformation that occurs at the surgical siteduring the performance of the surgical procedure; and is unable toprovide real-time imaging updates. In addition, real-timeintra-operative imaging surgical guidance techniques provides a limitedfield of view of the surgical site to the surgeon, and is unable toprovide comprehensive, global, whole-body anatomical information of apatient, which makes such surgical guidance techniques difficult to usein some instances. Furthermore, pathology/microscopy-based techniquesare unable to sample all surgical sites, and also requires a substantialamount of time to complete.

Therefore, there is a need for an imaging and display system for guidingmedical interventions, which provides surgical navigation,intra-operative medical guidance, and in-vivo microscopy medicalguidance (i.e. pathology surgical guidance), simultaneously, at the sametime. In addition, there is a need for an imaging and display system forguiding medical interventions, which provides a wearable display device,such as a wearable goggle-type display, for displaying the surgicalnavigation images, the real-time intra-operative images, and the in-vivoimaging/sensing (e.g. microscopy images or spectroscopy data) data atthe same time, to thereby provide an immersive, 3D, stereoscopic view,which imparts a natural sense of depth to the images viewed by thesurgeon wearing the display device. In addition, there is a need for animaging and display system for guiding medical interventions, whichprovides the ability to communicate via a communication network, so asto collaborate and share surgical guidance related images and any otherdata with any other computer device in communication with the network,such as smart phones, laptop computers and the like.

SUMMARY OF THE INVENTION

In a first embodiment, the present invention provides an imaging anddisplay system for guiding medical interventions comprising: a displayadapted to be worn by a user; a detector coupled to said display, saiddetector configured to capture intra-operative images from a target; anda computing unit coupled to said display and to said detector, saidcomputing unit adapted to store pre-operative images.

In a second embodiment, the present invention provides an imaging anddisplay system as in the first embodiment, wherein said display presentssaid pre-operative image and said intra-operative image simultaneously.

In a third embodiment, the present invention provides an imaging anddisplay system as in the first embodiment, wherein said display presentssaid pre-operative image and said intra-operative image simultaneouslyas a composite, co-registered image on said display.

In a fourth embodiment, the present invention provides an imaging anddisplay system as in the first embodiment, further comprising: acommunication interface coupled to said computing unit to enablecommunication with at least one other display.

In a fifth embodiment, the present invention provides an imaging anddisplay system as in the first embodiment, further comprising: aperipheral interface coupled to said computing unit, said peripheralinterface adapted to communicate with one or more peripherals.

In a sixth embodiment, the present invention provides an imaging anddisplay system as in the first embodiment further comprising: aperipheral interface coupled to said computing device, said peripheralinterface adapted to communicate with one or more peripherals, whereinsaid peripheral comprises a microscope (in vivo, hand-held orconventional) selected from the group consisting of: a fiber microscope,a handheld microscope, a color microscope, a reflectance microscope, afluorescence microscope, an oxygen-saturation microscope, a polarizationmicroscope, an infrared microscope, an interference microscope, a phasecontrast microscope, a differential interference contrast microscope, ahyperspectral microscope, a total internal reflection fluorescencemicroscope, a confocal microscope, a non-linear microscope, a 2-photonmicroscope, a second-harmonic generation microscope, a super-resolutionmicroscope, a photoacoustic microscope, a structured light microscope, a4Pi microscope, a stimulated emission depletion microscope, a stochasticoptical reconstruction microscope, an ultrasound microscope, andcombinations thereof.

In a seventh embodiment, the present invention provides an imaging anddisplay system as in the first embodiment, further comprising: aperipheral interface coupled to said computing device, said peripheralinterface adapted to communicate with one or more peripherals, whereinsaid one or more peripherals comprises a imaging system selected fromthe group consisting of: an ultrasound imager, a reflectance imager, adiffuse reflectance Imager, a fluorescence imager, a Cerenkov imager, apolarization imager, a radiometric imager, an oxygen saturation imager,an optical coherence tomography imager, an infrared imager, a thermalimager, a photoacoustic imager, a spectroscopic imager, a Ramanspectroscopic imager, a hyper-spectral imager, a fluoroscope imager, agamma imager, an X-ray computed tomography imager, an endoscope imager,a laparoscope imager, a bronchoscope imager, an angioscope imager, andan imaging catheter imager.

In an eighth embodiment, the present invention provides an imaging anddisplay system as in the first embodiment further comprising: aperipheral interface coupled to said computing device, said peripheralinterface adapted to communicate with one or more peripherals, whereinsaid peripheral comprises a spectrometer selected from the groupconsisting of: an optical spectrometer, an absorption spectrometer, afluorescence spectrometer, a Raman spectrometer, a coherent anti-stokesRaman spectrometer, a surface-enhanced Raman spectrometer, a Fouriertransform spectrometer, a Fourier transform infrared spectrometer(FTIR), a diffuse reflectance spectrometer, a multiplex orfrequency-modulated spectrometer, an X-ray spectrometer, an attenuatedtotal reflectance spectrometer, an electron paramagnetic spectrometer,an electron spectrometer, a gamma-ray spectrometer, an acousticresonance spectrometer, an auger spectrometer, a cavity ring down augerspectrometer, a circular dichroism auger spectrometer, a cold vapouratomic fluorescence auger spectrometer, a correlation spectrometer, adeep-level transient spectrometer, a dual polarization interferometry,an EPR spectrometer, a force spectrometer, a Hadron spectrometer, aBaryon spectrometer, a meson spectrometer, an nelastic electrontunneling spectrometer (IETS), a laser-induced breakdown spectrometer(LIBS), a mass spectrometer, a Mossbauer spectrometer, a neutron spinecho spectrometer, a photoacoustic spectrometer, a photoemissionspectrometer, a photothermal spectrometer, a pump-probe spectrometer, aRaman optical activity spectrometer, a saturated spectrometer, ascanning tunneling spectrometer, a spectrophotometry spectrometer,time-resolved spectrometer, a time-stretch spectrometer, a thermalinfrared spectrometer, an ultraviolet photoelectron spectrometer (UPS),a video spectrometer, a vibrational circular dichroism spectrometer, andan X-ray photoelectron spectrometer (XPS).

In a ninth embodiment, the present invention provides an imaging anddisplay system of as in the first embodiment, further comprising: aperipheral interface coupled to said computing device, said peripheralinterface adapted to communicate with one or more peripherals, whereinsaid peripheral comprises a tracking system selected from the groupconsisting of: an optical tracking system, an electromagnetic trackingsystem, a radio frequency tracking system, a gyroscope tracking system,a video tracking system, an acoustic tracking system, and a mechanicaltracking system.

In a tenth embodiment, the present invention provides an imaging anddisplay system as in the ninth embodiment, wherein the movement of saiddetector is configured to be tracked by said tracking system, such thatthe position of said intra-operative image captured by said detector isadjusted to maintain registration with said pre-operative image.

In an eleventh embodiment, the present invention provides an imaging anddisplay system as in the fifth embodiment, wherein said one or moreperipherals comprises a tracking system and an imaging or sensing probe,said probe capturing imaging or sensing data for composite presentationwith said intra-operative image and said pre-operative image on saiddisplay.

In a twelfth embodiment, the present invention provides an imaging anddisplay system as in the eleventh embodiment, wherein said probecomprises an in-vivo microscopy probe.

In a thirteenth embodiment the present invention provides an imaging anddisplay system as in the eleventh embodiment, wherein the movement ofsaid in-vivo microscopy probe is configured to be tracked by saidtracking system, such that the position of said probe is presented onsaid display.

In a fourteenth embodiment, the present invention provides the imagingand display system as in the first embodiment, wherein said displaycomprises a stereoscopic display.

In a fifteenth embodiment, the present invention provides an imaging anddisplay system as in the first embodiment, wherein said detectorcomprises a stereoscopic detector.

In a sixteenth embodiment, the present invention provides an imaging anddisplay system as in the first embodiment, wherein said display presentsa plurality of different imaging or sensing data in a picture-in-pictureformat.

In a seventeenth embodiment, the present invention provides an imagingand display system as in the first embodiment, wherein said detector isconfigured to detect one or more types of said intra-operative imagesselected from the group consisting of: a fluorescence image, areflectance image, a color image, a light absorption image, a lightscattering image, an oxygenation saturation image, a polarization image,a thermal image, an infrared image, a hyperspectral image, a light fieldimage, a fluorescence lifetime image, a bioluminescence image, aCerenkov image, a phosphorescence hyperspectral image, a spectroscopicimage, a chemilluminescence image and a scintillation image.

In an eighteenth embodiment, the present invention provides an imagingand display system as in the first embodiment, wherein saidpre-operative images comprise tomographic images.

In a nineteenth embodiment, the present invention provides an imagingand display system as in the first embodiment, wherein saidpre-operative images comprise 3D models processed from pre-operativetomographic data.

In a twentieth embodiment, the present invention provides an imaging anddisplay system as in the first embodiment, wherein said computing unitis configured to perform the steps comprising: computing atransformation matrices between a pre-operative image space, anintra-operative object/patient space and an intra-operative image space;and co-registering said pre-operative image space, said intra-operativeimage space and said intra-operative object/patient space.

In a twenty-first embodiment, the present invention provides an imagingand display system as in the first embodiment, wherein said computingunit is configured to perform the steps comprising: computing atransformation matrices between a pre-operative image space, anintra-operative object/patient space, an intra-operative image space anda peripheral image space; and co-registering said pre-operative imagespaces, said intra-operative image space, said peripheral image space,and said intra-operative object/patient space.

In a twenty-second embodiment, the present invention provides an imagingand display system as in the first embodiment, further comprising: alight source for illuminating said target.

In a twenty-third embodiment, the present invention provides an imagingand display system as in the twenty-second embodiment, wherein saidlight source comprises one or more white light-emitting diodes and oneor more band-rejection optical filters, wherein the frequencies of lightemitted by said light source that overlaps with a fluorescence emissionfrom said target is blocked by said band-rejection optical filters.

In a twenty-fourth embodiment, the present invention provides an imagingand display system as in the first embodiment, further comprising: alight source for illuminating said target, wherein said light sourcecomprises one or more projectors and one or more spectral filters.

In a twenty-fifth embodiment, the present invention provides an imagingand display system as in the first embodiment, further comprising: alight source wherein said light source comprise a pulsed illuminationdevice, or may utilize frequency modulation or pulse-durationmodulation.

In a twenty-sixth embodiment, the present invention provides an imagingand display system as in the first embodiment, further comprising: alight source, wherein said light source emits an illumination beam thatis provides an adjustable level of light frequencies that overlap withan emission spectra of said target.

In a twenty-seventh embodiment, the present invention provides animaging and display system as in the first embodiment, furthercomprising: a peripheral interface coupled to said computing unit, saidperipheral interface adapted to communicate with one or moreperipherals, wherein said peripherals comprise one or more trackingsystems, wherein said tracking systems comprise LEDs and spectralfilters.

In a twenty-eighth embodiment, the present invention provides an imagingand display system as in the first embodiment, further comprising: aperipheral interface coupled to said computing device, said peripheralinterface adapted to communicate with one or more peripherals, whereinsaid peripherals comprise one or more tracking systems, wherein saidtracking systems comprise software that enable topology sampling using atracked handheld imaging probe or a tracked handheld sensing probe

In a twenty-ninth embodiment, the present invention provides an imagingand display system as in the first embodiment, wherein said computingunit stores educational or medical training contents.

In a thirtieth embodiment, the present invention provides an imaging anddisplay system for guiding medical interventions comprising: a pluralityof goggles, each including: a stereoscopic display for viewing by theeyes of one wearing the goggle, a stereoscopic detector coupled to saidstereoscopic display, said detector having a field of view andprojecting an image within that field of view onto said display, and acommunication interface linking each of said plurality of goggles tocommunicate with each other.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features and advantages of the present invention willbecome better understood with regard to the following description,appended claims, and accompanying drawings wherein:

FIG. 1 is a perspective view of an imaging and display system forguiding medical interventions in accordance with the concepts of thepresent invention;

FIG. 2 is a schematic diagram showing the components of the imaging anddisplay system in accordance with the concepts of the present invention;

FIG. 3A is schematic diagram showing the components of a detectorprovided by the imaging and display system when configured withstereoscopic imaging sensors in accordance with the concepts of thepresent invention;

FIG. 3B is a schematic diagram of an alternative configuration of thedetector, whereby multiple sensor element types are used for each of thestereoscopic imaging sensors shown in FIG. 3A in accordance with theconcepts of the present invention;

FIG. 3C is a schematic diagram of another configuration of the detector,whereby multiple sensor element types are used for each of thestereoscopic imaging sensors shown in FIG. 3A in accordance with theconcepts of the present invention;

FIG. 3D is a schematic diagram of a further configuration of thedetector, whereby multiple sensor element types are used for each of thestereoscopic imaging sensors shown in FIG. 3A in accordance with theconcepts of the present invention;

FIG. 3E is a schematic diagram of another configuration of the detector,whereby multiple sensor element types are used for each of thestereoscopic imaging sensors used for each of the stereoscopic imagingsensors shown in FIG. 3A in accordance with the concepts of the presentinvention;

FIG. 4 is a schematic view of a composite image of a pre-operativesurgical navigation image, an intra-operative image, and an in-vivomicroscopy image that are simultaneously presented on the display of theimaging and display system for guiding medical interventions inaccordance with the concepts with the present invention;

FIG. 5 is a flow diagram showing the operational steps for a trackingand registration process in accordance with the concepts of the presentinvention;

FIG. 6 is a graph showing a plurality of illumination pulse patternsoutput by a light source for use with the imaging and display system inaccordance with the concepts of the present invention;

FIG. 7 is a graph showing another plurality of illumination pulsepatterns output by the light source in accordance with the concepts ofthe present invention;

FIG. 8 is a front perspective view of a shadowless surgical light inaccordance with the concepts of the present invention;

FIG. 9 is a general schematic view showing the use of a spectral filterwith individual lights of the shadowless surgical light of FIG. 11 inaccordance with the concepts of the present invention;

FIG. 10 is a general schematic view of a laser and laser diffuser lightsource, shown with the diffuser out of the path of the laser inaccordance with the concepts of the present invention; and

FIG. 11 is a general schematic view of a laser and laser diffuser lightsource, shown with the diffuser in the path of the laser in accordancewith the concepts of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

An imaging and display system for guiding medical interventions isgenerally referred to by reference numeral 100, as shown in FIG. 1 ofthe drawings. The system 100, shown in detail in FIG. 2, includes adisplay 110, which may comprise any suitable display, such as a wearabledisplay that is configured for being attached to and worn by a user 112.For example, the wearable display 110 may be included as part of agoggle-type wearable device 114 shown in FIG. 1, which comprises awearable goggle or eye-piece frame that carries the display 110.

In one aspect, the display 110 may comprise a single display elementsuitable for providing a single, continuous display that provides asingle display surface that encompasses the totality of the user's fieldof view, or portion thereof. Alternatively, multiple separate displayelements, may be used by the display, such as a dedicated right and adedicated left display, such as in the case of a stereoscopic display,which provides an independent displays, designated as 110A and 110B, asshown in FIG. 1, to provide the field of view of each user's eye.

Furthermore, the display 110 may comprise an LCD (liquid crystaldisplay) display, an OLED (organic light emitting diode) display, aprojection display; a head-mounted display (HMD), a head-mountedprojection display (HMPD), an optical-see through display, a switchableoptical see-through display, a selective occlusion see-throughhead-mounted display, and a video see-through display. Furthermore, thedisplay 110 may comprise an augmented reality window, augmentedmonitors, a projection on the patient/projective head-mounted display,selective occlusion see-through head-mounted display, and retinalscanning display. In another aspect, the display 110 may be configuredto display any static or moving image. The display 110 may also comprisea picture-in-picture (PIP) display that can display images from multipleindependent image sources simultaneously. In one example, the in vivomicroscopy image and intraoperative fluorescence image are displayed ina picture-in-picture fashion. In another example, the ultrasound imageand intraoperative fluorescence image are displayed in apicture-in-picture fashion. In another example, preoperative tomographicimages and intraoperative color images can also be displayed in apicture-in-picture fashion.

In one aspect, the display 110 may comprise a stereoscopic displaycapable of displaying stereoscopic images with depth perception. Inanother aspect, the display can be other type of 3D display capable ofdisplaying 3-dimensional images with depth perception. In still anotherembodiment, the display 110 may be configured to provide overlaid imagesof various opacity/transparency to allow simultaneous viewing ofmultiple images on the display 110 at one time. In yet anotherembodiment, the display 110 may be at least partially transparent toallow a user to view the image being displayed, while allowing the userto simultaneously see through the display 110 to also view the user'ssurrounding environment with natural vision at the same time.

Coupled to the display is a detector 120, which is used to acquireintra-operative images, which will be discussed in detail below. Itshould be appreciated that the intra-operative images acquired by thedetector 120 may be acquired and displayed at the display 110 inreal-time or near real-time. Specifically, the detector 120 isconfigured to capture any desired static or moving image data from atarget of interest (TOI) 130, which may comprise any desired object,such as a wound that is being treated in a patient 500, as shown inFIG. 1. That is, the detector 120 includes a field of view that capturesimage data of the target of interest 130 that is within the field ofview. It should also be appreciated that the detector 120 may be used inconjunction with any suitable optical lens or optical assembly toprovide any desired field of view, working distance, resolution and zoomlevel. In one aspect, the detector 120 may comprise a camera, such as acharge-coupled device (CCD), a complementary metal-oxide semiconductordevice (CMOS), one or more photomultiplier tubes (PMT), one or moreavalanche photodiodes (APD), photodiodes, and a thermographic camera,such as an infrared detector. In addition, the detector 120 may compriseone or more image intensifier tubes, a micro-channel plate imageintensifier, and a thin-film image intensifier.

In some embodiments, the detector is a single detector 120. In oneembodiment, the detector 120 may comprise a stereoscopic detector, whichincludes multiple imaging sensors or cameras designated respectively as120A and 120B, as shown in FIG. 1, which take stereoscopic images thatcan be displayed at stereoscopic display 110 with depth perception.

In another embodiment, the detector 120 may comprise a stereoscopicdetector, which includes multiple imaging sensors or cameras designatedrespectively as 120A and 120B, as shown in FIGS. 1 and 3A, whereby eachindividual camera 120A-B includes multiple individual sensor elements.For example, the cameras 120A-B may be each configured with a first andsecond sensor element, whereby the first sensor element provides forfull-color imaging and the second sensor element provides selective orswitchable florescence imaging. Further discussion of variousconfigurations of the various sensor elements that form the cameras120A-B will be discussed in detail below.

The detector 120 may be configured to perform one or more imaging modes,including but not limited to fluorescence imaging, thermal imaging,oxygen saturation imaging, hyperspectral imaging, photo acousticimaging, interference imaging, optical coherence tomography imagingdiffusing optical tomography imaging, ultrasound imaging, nuclearimaging (PET, SPECT, CT, gamma, X-ray), Cerenkov imaging, and the like.In addition, the detector 120 may also be configured to performreal-time/offline imaging, including absorption, scattering, oxygenationsaturation imaging, fluorescence imaging, fluorescence lifetime imaging,hyperspectral imaging, polarization imaging, IR thermal imaging,bioluminescence imaging, phosphorescence imaging, chemilluminescenceimaging, scintillation imaging, and the like.

The display 110 and the detector 120 are coupled to a computing unit200. The computing unit 200 may be part of a wearable version of thesystem 100 or might alternatively be an external computing unit 200. Thecomputing unit 200 includes the necessary hardware, software orcombination of both to carry out the various functions to be discussed.In one aspect, the computing unit 200 may comprise a microprocessor ormay comprise any other portable or standalone computing device, such asa smartphone, capable of communicating with the various components ofthe system 100. It should also be appreciated that the computing system200 may also include a memory module to store various data to bediscussed. In addition, the computing unit 200 is configured, wherebythe image data acquired by the detector 120 may be processed andtransmitted by the computing unit 200 in various manners to bediscussed. It should also be appreciated that the computing unit 200 mayinclude a local or remotely accessible memory or storage unit, whichallows the computing unit to store and/or acquire various programs,algorithms, databases, and decision support systems that enable avariety of functions to be discussed, which may be based on the imagedata collected by the detector 120. In one aspect the system 100 may bepowered by any suitable power source, such as a portable power sourcecomprising one or more batteries or a plug-in type power source forconnection to a standard electrical wall outlet.

In one aspect, the local or remote memory unit may store variouspre-operative image data, such as tomographic image data from MRIs andCT scans, which will be discussed in detail below.

In another aspect, the computing unit may perform image segmentation andgenerate a 3D model based on the preoperative imaging data. The 3D modelmay be stored in the local or remote memory unit.

In one aspect, the pre-operative image data, such as MRI (magneticresonance imaging) data for example, is segmented and processed into a3-dimensional model having a plurality of 3D surfaces. It should beappreciated that any suitable segmentation process may be used,including: either automatic, manual or semi-automatic segmentationprocesses. In addition, segmentation can also be based on thresholdingmethods, clustering methods, compression-based methods, histogram-basedmethods, edge detection methods, region-growing methods, split-and-mergemethods, partial differential equation-based methods, parametricmethods, level set methods, fast marching methods, graph portioningmethods, watershed transformation methods, model based segmentationmethods, multi-scale segmentation methods, trainable segmentationmethods, and any combination thereof.

In operative communication with the field of view of the detector 120 isa filter 150. Accordingly, the filter 150 serves to process the lightthat travels from the target of interest (TOI) 130 before the light isreceived at the detector 120 in the form of image data, such as imagedata for use during intra-operative imaging. As such, the filter 150 isconfigured to use any suitable technique to process the image datacollected by the field of view of the detector 120. In one aspect, thesystem 100 may be configured so that filter 150 is brought into or outof operative communication with the detector 120, so that the image datacollected by the field of view of the detector 120 is selectivelyfiltered or unfiltered. In one aspect, the selective filtering performedby the filter 150 may be carried out by any suitable mechanism, such asan electro-mechanical mechanism, which is initiated by any suitableswitching device 151, such as a mechanical switch, or voice command tomove the filter 150. Accordingly, when the switchable filter 150 is inoperative communication with the detector 120 the system 100 is placedinto a first mode for detecting TOIs 130 that emit frequencies within aspectrum of frequencies defined by the physical parameters of thefilter, such as the spectrum of frequencies emitted during thefluorescence of materials. Alternatively, when the filter 150 is not inoperative communication with the detector 120, the system 100 is placedinto a second mode for detecting TOIs 130 within another frequencyspectrum, such as the spectrum of frequencies for reflectance off theTOIs.

It should be appreciated that the filter 150 may comprise a filter wheelhaving different discrete filter of different filtering properties,which can be selectively rotated into operative alignment with thedetector 120. In addition, the filter 150 may comprise a long-passfilter, a band-pass filter, a tunable filter, a switchable filter, andthe like.

In another aspect, the filter 150 may comprise an 830 nm band-passfilter.

In other embodiments, the filter 150 may be replaced by a polarizer 152and operate in the same manner with respect to the detector 120 asdiscussed above with regard to the filter 150. Furthermore, in otherembodiments the polarizer 152 may be simultaneously used together withthe filter 150, whereby the field of view of the detector 120 isprocessed by both the polarizer 152 and by the filter 150 prior todetection by the detector 120. It should also be appreciated that thepolarizer 152 may comprise a switchable polarizer that operates in thesame manner as the switchable filter 150, or may comprise a tunablepolarizer.

Accordingly, the ability to selectively filter or selectively polarizethe field of view being detected by the detector 120 embodies a“convertible” system, whereby when the detector 120 is unfiltered, it isin a first mode, which is capable of a first imaging state, such asreflectance imaging; and when the detector is placed or “converted” intoits second mode, it is capable of a second imaging state, whereby it iscapable of fluorescence imaging for example.

Furthermore, using the combination of the cameras 120A-B each havingmultiple imaging elements together with the selective use of the filter150 or polarizer 152 allows for a variety of modes of operation. Forexample, in FIGS. 3B-D the detector 120 is configured such that eachcamera 120A and 120B has two sensor elements 122 and 124, whereby thefirst sensor element 122 is used for a first imaging mode (or aconvertible detection mode that is switchable between among two or moreimaging modes) and the second sensor element 124 is used for a secondconvertible imaging mode, which provides selective imaging among two ormore imaging modes. Thus, in FIG. 3B, sensor element 122 of cameras120A-B are operate in a color imaging mode, while sensor elements 124 ofcameras 120A-B operate in a convertible filter mode, that can beswitched between florescence imaging with different spectralfrequencies; or between polarization imaging with different polarizationstates. In addition, FIG. 3C shows that the sensor element 122 ofcameras 120A-B is switchable between different modes of fluorescenceimaging, while sensor element 124 of cameras 120A-B are switchablebetween different modes of polarization imaging. Furthermore, FIG. 3Dshows that the sensor element 122 of cameras 120A-B is a thermographicsensor, while sensor element 124 of cameras 120A-B are switchablebetween different modes of fluorescence imaging; or switchable betweendifferent modes of polarization imaging. Additionally, FIG. 3E shows theuse of three sensor elements, whereby sensor element 124 of cameras120A-B offer a first-type of fluorescence imaging modes; sensor element122 of cameras 120A-B is offer color imaging or thermographic imaging;and the sensor element 126 of cameras 120A-B offers a second-type offluorescence imaging modes.

Coupled to the computing system 200 is a communication interface 250,which includes a suitable antenna 252 for communicating wirelessly orvia a wired connection with a communication network 260. The system 100may communicate via the communication network 260 with other imaging anddisplay devices 100A-X, or any other networked computer system 262, suchas laptop computers, smart phones, and the like, as shown in FIG. 1. Inone aspect, the communication interface 250 is embodied as a transceiverthat is enabled to both transmit and receive data via the network 260.In one aspect, the communication interface 250 may be configured tocommunicate over the network 260 using any suitable method, including RF(radio frequency) signals, such as a low-power RF signals, a wired orwireless Ethernet communication method, a WiFi communication method, aBluetooth communication, cellular communication (e.g. 3G, 4G, LTE, etc.)and the like. As such, the ability of multiple systems 100 tocommunicate with each other enables a variety of functions, which willbe discussed in detail below.

The communication interface 250 also enables network and cloud computingfeatures to be carried out by the imaging and display system 100. In oneaspect, the communication interface 250 allows the system 100 tocommunicate with a remote storage devices on a remote network or aremote cloud computing system, generally represented by the numeral 270,as shown in FIG. 1 to allow access centralized data storage, conductfurther computing analysis, access to other software applications, andto enable record storage. For example, the system 100 may acquirepre-operative images from the remote network 270.

Also coupled to the computing device 200 is a peripheral interface 300.The peripheral interface may comprise a wired or wireless interface thatallows for the addition of one or more peripherals 350 to be selectivelyadded to the imaging and detection system 100. The peripherals maycomprise one or more sensors and detectors. For example, such add-onperipheral 350 may include a vital sign sensor module, that may monitorone or more of: temperature, blood pressure, pulse, respiratory rate,ECG, EEG, pulse oximetry, blood glucose, and the like. The peripheral350 may also include an ultrasound module, a spectroscopy module (e.g.Raman spectroscopy, absorption spectroscopy, and reflectancespectroscopy), a GPS (global positioning system) module, a microscopemodule (e.g. a handheld microscope, a fiber-based in-vivo microscope,and a traditional microscope), and a non-microscopic imaging module(hyperspectral imaging, photoacoustic imaging, optical coherenceimaging).

In another aspect, the peripheral 350 may comprise a probe instrument,such as a hand-held probe used to acquire or sense any in-vivo target ofinterest 130. As such, the hand-held probe may be used for any desiredtype of microscopy, such as in-vivo microscopy. The probe may utilizevarious detection methods, such as color microscopy, reflectancemicroscopy, fluorescence microscopy, oxygen-saturation microscopy,polarization microscopy, infrared microscopy, interference microscopy,phase contrast microscopy, differential interference contrastmicroscopy, hyperspectral microscopy, total internal reflectionfluorescence microscopy, confocal microscopy, non-linear microscopy,2-photon microscopy, second-harmonic generation microscopy,super-resolution microscopy, photoacoustic microscopy, structured lightmicroscopy, 4Pi microscopy, stimulated emission depletion microscopy,stochastic optical reconstruction microscopy, ultrasound microscopy,and/or a combination of the aforementioned, and the like.

In another aspect, the handheld probe used as the peripheral 350 may bea imaging device that has not reached microscopic resolution yet. Insome embodiments, the non-microscopic imaging method is selected fromone or more of the following: reflectance imaging, fluorescence imaging,Cerenkov imaging, polarization imaging, ultrasound imaging, radiometricimaging, oxygen saturation imaging, optical coherence tomography,infrared imaging, thermal imaging, photoacoustic imaging, spectroscopicimaging, hyper-spectral imaging, fluoroscopy, gamma imaging, and X-raycomputed tomography. The physical form of the handheld probe maycomprise an endoscope, a laparoscope, a bronchoscope, an angioscope, anda catheter for angiography.

In still another example, the handheld probe may be a non-imaging deviceor a sensing device, such as a fiber-based spectrophotometer. Inaddition, different spectroscopies may be realized by the peripherals350, such as various optical spectroscopies, absorption spectroscopy,fluorescence spectroscopy, Raman spectroscopy, Coherent anti-StokesRaman spectroscopy (CARS), surface-enhanced Raman spectroscopy, Fouriertransform spectroscopy, Fourier transform infrared spectroscopy (FTIR),multiplex or frequency-modulated spectroscopy, X-ray spectroscopy,attenuated total reflectance spectroscopy, electron paramagneticspectroscopy, electron spectroscopy, gamma-ray spectroscopy, acousticresonance spectroscopy, auger spectroscopy, cavity ring downspectroscopy, circular dichroism spectroscopy, cold vapour atomicfluorescence spectroscopy, correlation spectroscopy, deep-leveltransient spectroscopy, dual polarization interferometry, EPRspectroscopym, force spectroscopy, Hadron spectroscopy, Baryonspectroscopy, meson spectroscopy, Inelastic electron tunnelingspectroscopy (IETS), laser-induced breakdown spectroscopy (LIBS), massspectroscopy, Mossbauer spectroscopy, neutron spin echo spectroscopy,photoacoustic spectroscopy, photoemission spectroscopy, photothermalspectroscopy, pump-probe spectroscopy, Raman optical activityspectroscopy, saturated spectroscopy, scanning tunneling spectroscopy,spectrophotometry, time-resolved spectroscopy, time-stretchSpectroscopy, thermal infrared spectroscopy, ultraviolet photoelectronspectroscopy (UPS), video spectroscopy, vibrational circular dichroismspectroscopy, X-ray photoelectron spectroscopy (XPS), or a combinationof the aforementioned.

Tracking and Registration of Multiple Images for Display

In some embodiments, the system 100 includes a tracking module, whichcan be considered another peripheral 350, and includes software suitablefor tracking the spatial location of the patient, location of thedetector 120 (or 120A, 120B) and the location of peripherals 350, suchas imaging cameras and probes, and registering these locations relativeto the image(s) of the detector 120 or detectors 120A, 120B (instereoscopic modalities). Reference to detector 120 herein will also beunderstood to be equally applicable to the stereoscopic modalities ofthose systems 100 employing detectors 120A and 120B. Furthermore,through tracking the position of the patient and the position of thedetector 120 (or 120A, 120B), the preoperative images (such as CT, MRI,SPECT, PET images) can be registered to the image(s) of the detector 120or detectors 120A, 120B (in stereoscopic modalities). Thus, thecorresponding imaging and sensing information obtained from theperipheral 350, and the preoperative images, can be correlated with thefield of view imaged by the detector 120 of the imaging and displaysystem 100. That is, the system 100 may be programmed to utilizetracking and registration techniques to allow for the overlay ofmultiple images acquired directly by the detector 120 of the system 100,with preoperative images, and with those images acquired by peripheralimage detectors, such as hand-held microscopy probes, or the like. Insome embodiments, the tracking module can also track and register thelocation of other non-peripheral elements, such as the tools beingemployed by military or medical personnel. For example, the location ofscalpels or clamps or stents or other elements of a medical operationcould be tracked and registered with the images. It should beappreciated that the software enabling such tracking and registrationfeatures may be provided from a remote computer system to the system 100via the network 260 or stored on any peripheral attached to theperipheral interface 300. Specifically, tracking techniques utilized bythe system 100 obtain the position of a patient to be treated by thesystem 100, the system 100 itself comprising the wearable display 114,and the handheld imaging peripheral 350 coupled to the peripheralinterface 300.

It should also be appreciated that the peripheral module 350 may includea tracking module, which allows the spatial location of the detector120, and spatial location of plug-in modules within the peripheral 350,such as imaging cameras and probes. Thus, the corresponding imaging andsensing information obtained from the peripheral 350 can be correlatedwith the field of view imaged by the detector 120 of the imaging anddisplay system 100. That is, the system 100 may be programmed to utilizetracking and registration techniques to allow for the overlay ofmultiple images acquired directly by the detector 120 of the system 100with those images acquired preoperatively (such as CR, MRI, SPECT, PET,X-ray, Gamma imaging, etc). Alternatively, the system 100 may beprogrammed to utilize image tracking and registration techniques toallow for the overlay of multiple images acquired directly by thedetector 120 of the system 100 with those 3D models constructed based ondata acquired preoperatively (such as CR, MRI, SPECT, PET, X-ray, Gammaimaging, etc). Furthermore, the system 100 may be programmed to utilizeimage tracking and registration techniques to allow for the overlay ofmultiple images acquired directly by the detector 120 of the system 100with those images acquired (i.e. imaged/sensed) by peripheral imagedetectors, such as hand-held in-vivo microscopy probes 350, or the like.Similarly, multiple images acquired directly by the detector 120,preoperative images or 3D models, images acquired (i.e. imaged/sensed)by peripheral image detectors 350 can be overlaid and registeredtogether. It should be appreciated that peripheral sensing detectors 350data may also be registered with multiple images acquired directly bythe detector 120, preoperative images or 3D models. It should be furtherappreciated that the picture-in-picture display and overlaid display canbe used in conjunction with each other, in a hybrid mode. It should beappreciated that the software enabling such tracking and registrationfeatures may be provided from a remote computer system, such as remotesystem 270, to the system 100 via the communication network 260 orstored on any peripheral attached to the peripheral interface 300.Specifically, tracking techniques utilized by the system 100 obtain theposition of each of the following: the position of the patient to betreated using the system 100, the position of the system 100 itselfcomprising the wearable system including the display 110 and detector120, and the position of the handheld imaging/sensing peripheral 350,such as in-vivo probe, coupled to the peripheral interface 300.

Furthermore, the tracking functions may be carried out using opticaltracking or magnetic tracking devices that are employed as a peripheral350. If optical tracking is used, active markers such as LEDs may beattached to detector 120, the imaging or sensing probe employed asanother peripheral 350 and the patients, to locate their locations,respectively. NDI Optotrak Certus system is an example of opticaltracking systems that may be used for this embodiment. Commerciallyavailable optical tracking systems may consist of CCD cameras andsequentially illuminated infrared (IR) LEDs, and can be easilyintegrated as a peripheral 350, into the wearable imaging and displaydevice 100. Alternatively, one may use a videometric system to estimatepatient pose and instrument orientation by identification of passivemarkers in video-image sequences.

In one aspect, optical tracking using NDI Optotrak Certus mayincorporated as a peripheral 350 to provide tracking, whereby lightemitting diodes (LED) are attached to the wearable device 100 thatcarries the detector 120, and imaging module as another peripheral 350,such as ultrasound and hand-held microscopy probes and patients. Assuch, the LEDs attached to the detector 120, hand-held probe 350, andpatients are tracked by the NDI Optotrak Certus system.

In another embodiment, a novel infrared optical tracking method may beutilized by the system 100. As such, the wavelength of the opticalemitters for tracking purposes (such as LEDs) attached to the patient,wearable imaging and display system 100, and intraoperative imagingperipheral 350, may be different wavelengths from the wavelengthsdetected by the detector 120, and imaging peripheral 350. Methods, suchas spectral filtering may be used to facilitate the separation ofwavelengths between the optical emitter from the tracking system and thedetection of the detector 120, and imaging peripheral 350. Frequencymodulation may also be used to separate the signal from the trackingoptical emitters from the signal-of-interest of the detector 120, andimaging peripheral 350.

In another example, gyroscopic tracking in combination with videotracking may be performed using the module 350.

If electromagnetic tracking is used, the peripheral 350 may incorporatesmall coils or similar electromagnetic field sensors and multipleposition measurement devices. The electromagnetic field sensors may beattached to detector 120, the imaging or sensing probe employed asanother peripheral 350 and the patients, to locate their locations,respectively.

Alternatively, the tracking functions may be carried out using fiducialmarkers, such as LEDs, attached to the patient to be treated, thewearable imaging and display device 100, and the imaging peripheral 350.With the position obtained using the tracking techniques described,enabled by tracking systems as a peripheral 350, registration, oralignment, of the different images obtained by the imaging and displaydevice 100 and the handheld imaging peripheral 350 is performed by usingtransformation matrices between preoperative imaging space,intraoperative object space (i.e patient space), intraoperative imagingspace (device 100 imaging space), and the handheld peripheralimaging/sensing probe 350 space (i.e. peripheral imaging probe space)can be calculated. Specifically, the image registration process iscarried out such that preoperative imaging space can be registered tointraoperative object space (patient space); intraoperative imagingspace (device 100 imaging space) can be registered to intraoperativeobject space (patient space); and handheld device imaging space(peripheral 350 eg. Ultrasound, fiber microscope, etc.) can beregistered to intraoperative object space (patient space). As a result,the co-registered intra-operative images obtained from the detector 120of the wearable system 100 and the in-vivo images acquire from thein-vivo probe peripheral 350, and pre-operative tomography images can bedisplayed in the wearable display in an overlaid and aligned manner.

Thus, in some embodiments, the tracking functions may be carried outusing fiducial markers, such as LEDs, attached to (a) the patient to betreated or an object to be acted upon or observed (in the instance ofnon-medical applications), (b) the wearable imaging and display device100, and (c) the peripheral 350. Through the use of fiducial markers,images of the same subject produced with multiple distinct imagingsystems—for example, the detector 120 as a first imaging system, and anydesired peripheral 350 that generates a second image as the secondimaging system—may be correlated by placing fiducial markers in the areaimaged by both systems. Appropriate software correlates the two images,and in the case of the present invention, permits viewing of the two (ormore) images overlaid together or in a picture-in-picture format.

It should be appreciated that during the tracking and registrationprocesses the computing unit 200 carry out computations and executesoftware to enable the accurate tracking and registration. In oneaspect, the complete registration processes are represented by aflowchart is shown in FIG. 5 designated by number 600. Generally, theprocess 600 obtains the position of the patient 500, the wearableimaging and display system 100, and the handheld probe 350. Initially,at step 602, the system 100 acquires pre-operative imaging data. Next,at step 604 a 3D model is created based on the pre-operative imagingdata. At step 606 the position of the patient is trackedintra-operatively using any suitable technique, such as fiducial markersfor example. At step 608 a transformation matrix is calculated betweenthe pre-operative image space and the intra-operative object space (i.e.patient space). Continuing, the pre-operative image data is registeredto the intra-operative object space (i.e. patient space), as indicatedat step 610. At step 612, the intra-operative imaging data is acquiredfrom the wearable imaging and display system 100, such as fluorescenceor color imaging for example. Continuing to step 614, the position ofthe wearable imaging and display system 100 is obtained, using anysuitable technique, such as optical tracking or magnetic tracking). Atstep 616, calculate the transformation matrix between theintra-operative imaging space (i.e. wearable imaging and display system)and the intraoperative object space (patient space). Next, at step 618register the intra-operative imaging space (such as fluorescence imagedata) to the intra-operative object space (i.e. patient space). At step620, the process 600 acquires handheld device imaging or sensing data,such as ultra-sound fiber microscope, and Raman spectroscopy forexample. Next, at step 622 the position of the hand-held probe 350, suchas an ultrasound fiber, a microscope, and Raman spectroscopy probe istracked. At step 624 a transformation matrix is calculated between thehand-held imaging/sensing probe 350 image space and the intra-operativeobject space (i.e. patient space). Next, at step 626, the hand-helddevice image space (i.e. ultrasound or microscope) is registered to theintra-operative object space (i.e. patient space), as indicated at step626. Finally, at step 628 the co-registered image data is presented onthe display 110 of wearable imaging and display unit 100.

In another aspect, the process 600 may be configured, such that thetracking and registration process is performed without the image dataacquired from the hand-held probe 350. As a result, the process 600 onlyuses the intra-operative image data acquired by the imaging and displaysystem 100 (i.e. goggle system) and the pre-operative surgicalnavigation image data. In such a case, only steps 602-618 and step 628of the process 600 are required to be performed.

In yet another aspect, the process 600 may also be configured, such thatthe tracking and registration process is performed without thepre-operative surgical navigation image data. As a result, the process600 only uses the intra-operative image data acquired by the imaging anddisplay system 100 (i.e. goggle system) and the image data acquired bythe hand-held probe 350. In such a case, only steps 612-626 and step 628are performed.

It should also be appreciated that in addition to the trackingtechniques described above, other tracking techniques may be used, suchas radio frequency tracking, gyroscope tracking, video tracking (patternrecognition), acoustic tracking, mechanical tracking, and/or acombination thereof. In addition, the tracking method employed by themodule 350 may utilize rigid body, flexible body or digitizer methods.

It should also be appreciated that in addition to the registrationtechniques discussed above, other registration techniques may be used,such as point-based registration, surface-based registration, and/or acombination thereof. The registration may comprise eitherintensity-based or feature-based registration. The transformation modelsused may comprise linear transformation, or non-rigid/elastictransformation. Spatial or frequency domain methods may be used, as wellas automatic or interactive methods.

To sample the topology of the object space in the field of view (or thetarget of interest), digitizers (such as the device from NDI) may beused to sample the points in object space. Alternatively, topologyacquisition systems, such as a 3D scanner may be used to capture the 3Dtopology, which may facilitate image registration.

The handheld probe employed as a peripheral module 350 may serve dualpurposes: serving as stylus/digitizer for sampling topology; and servingas imaging or sensing probe. Specifically, the handheld probe may haveoptical emitters such as LEDs attached to it, which will allow locationof the tip of the handheld probe with the help of the optical trackingsystem; Alternatively, the position of the tip can be obtained bytracking the electromagnetic sensors attached to the handheld probeusing a magnetic tracking system. When the probe are swiped acrossdifferent points on the surface of the organs, a 3D point cloud can beestablished, based on the locations of the tips of handheld probe (tipis considered to be just in contact with organs). In this way, theimaging handheld probe also enables similar functionality to sampletopology as the non-imaging stylus/digitizer traditionally employed intracking systems.

In another aspect, a tracker module employed as a peripheral module 350may track the positions of a hand-held microscopy probe peripheral alsoemployed as a peripheral module 350, register each image with thecorresponding spatial counterpart in the preoperative image, and displayin the display 110. As such, the images detected by the imagingperipherals, such as a ultrasound probe may then be overlaid with imagescollected, such as fluorescence images, by the detector 120 of theimaging and display system 100 for presentation on the display 110. Theregistration of multiple images on the display 110 may be achieved usingany suitable technology, including point-based registration,surface-based registration, intensity-based, feature-based registration,and/or a combination of both. The transformation models used maycomprise linear transformation, or non-rigid/elastic transformation.Spatial or frequency domain methods may be used, as well as automatic orinteractive methods. For example, fiducial markers, such as LEDs, may beused to facilitate point-based registration. In another example, ifsurface topology or profile is available, the surface-based registrationcan also be used. In yet another example, the registration may also bebased on pattern recognition or feature-based recognition.

Thus, by combining the functionality of the communication interface 250and the peripheral interface 300, the system 100 is enabled to providemultiple functions. One or more peripherals of a multitude of types,including those mentioned above can be selectively coupled to thedisplay system 100, as needed for providing the system 100 with adesired functionality. If imaging from a probe is needed in a givenapplication, for example for in vivo imaging of a patient, a probe as aperipheral 350 can be coupled to the display system 100 at the interface300 so that the display system 100 would then have the ability todisplay the image gathered from the probe. As per the trackingdisclosure above, this image could be overlaid onto the image of thepatient gathered by the detector 120, placing the in vivo image of theprobe at the proper location on the image of the patient. Furthermore,the probe image can be overlaid onto the preoperative images,intraoperative images captured by detector 120, placing the in vivoimage of the probe at the proper location on the image of the patient.In one aspect, the handheld microscopy probe can scan over a larger areaover the patient sequentially. Using tracking technique discussed above,the small field of view microscopy image captured by 350 may be joinedtogether as a montage, which is overlaid over intraoperative imagecaptured by 120 and preoperative image.

In another aspect, a co-registration of a 4 sensor setup between colorand fluorescence imaging, whereby vertical and horizontal disparitiesare correlated. In particular, this example describes the manner inwhich a 4-camera setup is used to register intraoperative color imagingto intraoperative fluorescence imaging.

In another embodiment, stereoscopic fluorescence images captured by 2fluorescence cameras and stereoscopic color images captured by 2 colorcameras can be registered together. Both sets of images were placed intoside-by-side frames, and the fluorescent side-by-side frame was overlaidonto the anatomical frame by the computing module and sent to thedisplay. For high registration accuracy, we measure the verticaldistance from the center of the filtered cameras for fluorescence to thecenter of the unfiltered color camera as well as the horizontal baselinedistance between two filtered or unfiltered cameras. From thisinformation, a correction metric, D_(V), was determined from theequation:

$\frac{L_{H}}{L_{V}} = \frac{D_{H}}{D_{V}}$

where L is the measured baseline disparity between cameras in either thehorizontal (H) or vertical (V) direction, and D_(H) is the horizontalpixel disparity between common points in the left and right fluorescentimages. The points used to calculate D_(H) were the peak fluorescentpoints; if more than one peak existed, one was chosen for thecalculation. The fluorescent frames were then shifted up by thecalculated correction metric so that, after calibration, the fluorescentimage was aligned to the corresponding color image.

In addition, GPS and wireless communication between multiple imaging anddisplay systems 100A-X can be integrated, such that information relevantto medical environments is labeled with GPS data. Thus, in oneembodiment, information acquired by each system 100A-X can also betransmitted or received wirelessly, to guide medical interventions.Using telemedicine functionality of the system 100, medical operationscan be performed by first responders using the system 100 under theguidance of medical practitioners that are located remotely but who arealso using the system 100. It should be appreciated that the systems 100A-X may also communicate with any other suitable computing device, suchas a tablet, mobile smart phone, or the like.

In addition, the system 100 may include an illumination or light source400 to illuminate the field of view used to image the target object ofinterest 130 being imaged by the detector 120. It should also beappreciated that the light source 400 is configured to deliver a lighthaving the appropriate intensity and frequency spectrum that iscompatible with the particular imaging being conducted with the detector120, with or without the filter/polarizer 150,152. For example, it maybe necessary to have a light source 400 that emits a first frequencyspectrum for use in a first imaging mode, such as color reflectanceimaging mode, and that emits a second frequency spectrum for use in asecond imaging mode, such as a fluorescence imaging mode. In one aspect,the light source 400 may be coupled to the computing device 200 forautomated control over the functions provided by the light source 400,or may be unattached from the computing device 200 and operated manuallyby the user of the system 100.

It should also be appreciated that the light source 400 may servedifferent purposes in the medical environment. Furthermore, uponconversion of the detector 120 by removal or the filter/polarizer150,152 or by selecting the necessary filter/polarizer 150,152 theillumination of the light source 400 may be used for florescenceimaging, optical imaging, photodynamic therapy, laser surgery,sterilization, and the like. It should also be appreciated that multiplelight sources 400 may be used.

It should also be appreciated that the light source 400 may comprise alaser light; a light emitting diode (LED), such as a white LED; anincandescent light; a projector lamp; an arc-lamp, such as xenon, xenonmercury, or metal halide lamp; as well as coherent or in-coherent lightsources.

The light source 400 may also comprise a digital (LED-based) projector,and additionally the light source may project spatial frequencies forpatterned illumination. For example, a digital projector in conjunctionwith spectral filters may be used as the light source. In addition, thelight source 400 may emit a continuous or pulsed output, and maygenerate light that is within any desired spectral window ofelectromagnetic waves. It should also be appreciated that the lightsource 400 may also include a light diffuser.

Spectral Filter to Block Light Frequencies Overlapping with FluorescenceEmission Spectra

In some embodiments, particularly when it is desired to observe afluorescence emission spectra from the object being illuminated andobserve through the imaging and display system 100, the light source 400selectively shines through a spectral filter 402, as shown in FIG. 2,that blocks the wavelength of the emission spectra to be observed, suchthat the light source 400 does not interfere with the observance of thatemitted wavelength. For example, if the object is to be observed forfluoresce at a certain wavelength, the spectral filter 402 would bechosen to block that wavelength from the light source so that the lightsource does not interfere with the observance of the emittedfluorescence. In some such embodiments, the light source is a whitelight source thus providing a broad spectrum, and the spectral filter isappropriately chosen based on the emission spectra to be observed. Insome embodiments, the light source is one or more white light emittingdiodes (LED). In some embodiments, the individual light sources arewhite light emitting diodes (LED) that are filtered by a 775 nm low-passfilter. In another embodiment, the low-pass filter may be replaced witha polarizer, or may be used in conjunction with the filter the lightsource shines through a spectral filter.

With reference to FIGS. 8 and 9, in another embodiment, the light source400 may comprise a shadow-less light 404, which is desirable for useduring surgery (i.e. a surgical light). The shadow-less light 404includes a plurality of individual light sources 406 spaced apart in asupport 407 to project light onto an object, such as patient 500,whereby a shadow cast by an intervening object and one or more of theplurality of individual light sources is negated by at least one otherof the plurality of individual light sources. For example, theshadow-less light 404 can be a surgical light and a surgeon my interposea hand and arm between the shadow-less light 404 and the patient andthus certain individual light sources would tend to cast a shadow ontothe patient but for the fact that other light sources will not have thehand/arm of the surgeon interposed between the shadow-less light sourceand the surgeon such that those lights will negate the shadow, thusleading to shadow-less lighting. As known, the support 407 is on the endof a swing arm 410, or a goose neck or other connection providing theability to position the light 404 as desired.

In some embodiments, particularly when it is desired to observe anemission spectra from the object, the individual light sources 406 ofthe shadow-less light 404 selectively shine through a spectral filter408 (FIG. 9) that blocks the wavelength of the emission spectra to beobserved, such that the shadow-less light source does not interfere withthe observance of that emitted wavelength. In some embodiments, theindividual light sources are white light emitting diodes (LED). In someembodiments, the individual light sources are white light emittingdiodes (LED) that are filtered by a 775 nm low-pass filter. In anotherembodiment, the low-pass filter may be replaced with a polarizer, or maybe used in conjunction with the filter.

In a particular embodiment, the light source 400 is afluorescence-friendly shadow-less surgical light, which can providewhite light surgical illumination and florescence illumination withoutleaking frequencies overlapping with fluorescence emission. Thisshadow-less light offers both well-rendered surgical illumination (lookslike white light to naked light) and fluorescence excitation at the sametime. In one embodiment, such light source comprises a plurality ofwhite light emitting diodes (LED) coupled with Notch Filters that areOptical Filters that selectively reject a portion of the spectrum, whiletransmitting all other wavelengths. With the notch the frequenciesoverlapping with fluorescence emission, which are emitted by white LEDs,are rejected. It should be appreciated that in some cases edge filterscan be used to achieve similar results in blocking the frequenciesoverlapping with fluorescence emission. In one example, the shadow-lesslight source comprises a plurality of white light emitting diodes (LED)that is filtered by a 775 nm low-pass filter. It should be appreciatedthat thin films or other devices may play similar role as notch filtersor edge filters in the fluorescence-friendly shadow-less surgical light.In one aspect, the shadow-less light 400 may comprise an array of whitelamps with edge filters or notch filters. In another embodiment, thespectral filters may be replaced with polarizers, or may be used inconjunction with the filters.

In some embodiments, the light source is a traditional projector (lampbased) or digital projector (LED-based) selectively used in conjunctionwith spectral filters or polarizers (as described with other lightsources). The projector can also selectively project spatial frequencies(i.e., provide patterned illumination). The spectral filters can be in afilter wheel as already described. The projector beneficially provides awell-defined illumination area. The projector can be set to project anydesired wavelength of light and can project without brighter and dimmerareas (i.e., provides consistent light).

With reference to FIGS. 10 and 11, in another embodiment, the lightsource 400 comprises a laser diode 412 and a diffuser 414 movable to beselectively interposed between the laser diode 412 and the object.Without the diffuser 414 interposed, the laser diode 412 simply shines afocused beam of light, while, with the diffuser 414 interposed, thelaser shines over a greater surface area and is suitable for generalillumination. In some embodiments this can allow for switching betweenlaser aiming and night vision (with diffuser out of light path) orfluorescence-guided treatment (with diffuser in light path). Inaddition, the laser diode with diffuser 400 may also use a filter. Inaddition, the laser diode 400 may also be pulsed, or frequency modulatedto reduce the average amount of light energy delivered.

Pulsing of Light

As seen in FIGS. 9 and 10, in some embodiments, the light source 400 maycomprise a pulsed light source, or may utilize frequency modulation orpulse-duration modulation. In one aspect, the detector 120 may detectsignals of a given frequency or spectrum, and the light source 400 maycorrelate the detected signal with the frequency modulation andpulse-duration modulation. In one aspect, the light source 400 maymodulate the emitted light using an electro-optic modulator, opticalchopper, or the like. Alternatively, if the light source 400 comprisesone or more light emitting diodes (LED) the light source 400 may operateto adjust the intensity of light being output by adjusting the frequencyof the AC (alternating current) that is supplied to power the LEDs.

Specifically, as shown in FIG. 6, the DC component of the light source400 detected by the goggle system 100 are the fluorescence image type-2,and the AC component of the light detected by the goggle system 100 areflorescence image type-2. The goggle system 100 may use a 2-camera setupor a 4-camera setup. The goggle system 100 is configured to detect thesignals, correlated with the frequency modulation or pulse-durationmodulation. Various ways of modulating the light may be used, such as anelectro-optic modulator, an optical chopper, or the like. If LEDs areused, the illumination output by the light source 400 can be modulatedby supplying AC current of desirable frequency through the LEDs. Alock-in amplifier may be used by the system 100. It should beappreciated that light bulbs, lamps, laser diodes, lasers or the likecould be used instead of LED based light source 400.

Furthermore, as shown in FIG. 7, the frequency component of the lightsource 400 designated f1, which is detected by the goggle system 100will be the fluorescence image type-1, and the frequency component ofthe light designated f2 that is detected by the goggle system 100 is thefluorescence image type-2. The goggle system 100 may use a 2-camerasetup or 4-camera setup, and the goggle system 100 will detect thesignals, correlated with the frequency modulation or pulse-durationmodulation. Possible ways of modulating the light may compriseelectro-optic modulator, optical chopper, or the like. In addition, ifLEDs are used, the illumination output by the light source 400 can bemodulated by supplying AC current of desirable frequency through theLEDs. In addition, a lock-in amplifier may be used by the system 100. Itshould be appreciated that light bulbs, lamps, laser diodes, lasers orthe like could be used instead of LED based light source 400.

It is also contemplated that the system 100 includes a microphone 480and a speaker 490 to enable verbal communication between the varioussystems 100A-X and other computer systems (i.e. tablet computers, smartphones, desktop computers), and the like using the communication network260.

Thus, with the structural arrangement of the various components of theimaging and display system 100 set forth above, the following discussionwill present various embodiments of the system 100 for executingspecific functions.

Polarization

The system 100, as previously discussed by use the polarizer 152 in aconvertible or selective manner, such that when polarization is invokedin a first state, the detector 120 provides polarization-gated imaging,polarization difference imaging, spectral-difference polarizationimaging, Muellar matrix imaging.

For example, the system 100 may also use traditional division of timetechniques, as well as tunable liquid crystal polarization filters ordivision of focal plane technology (e.g. Moxtek micropolarizer arrays).

Networked System

As previously discussed, each imaging and display system 100 includesthe detector 150 and a communication interface 250, which allows aplurality of systems 100A-X to communicate various data with one anotherand/or with one or more remote computing devices. It should beappreciated that the system 100 may be configured to form ad-hocnetworks between each one of the individual systems 100A-X, or may beconfigured to join any exiting wireless communication network, such as acellular data network, radio-frequency communication, wireless LAN,wireless PAN, WiFi or Bluetooth network for example. As previouslydiscussed, each system 100 has the ability to be a sender of data and arecipient of data. It should be appreciated that system 100 can senddata to any type of display unit, including other non-wearable displayand wearable display units, to enable visualization of content displayedat system 100.

In one embodiment, the detector 120 of one system 100 may capture imageor video data that is transferred over the network to one or more othersystems 100A-X or any other computing device (i.e. tablet, computer,smartphone) that are connected to the communication network. Such imagetransfer may occur simultaneously between the systems 100A-X inreal-time or in near real-time. The real-time or near real-timetransmission of image or video data, such as viewing an patients, fromone system 100 may be used by recipients of the image or video data atone or more other users of the system 100, or any other users ofwearable displays, or any other users of other computer systemsconnected to the network, in order to analyze and provide medicalguidance based on the transferred images. In addition, such networkedsystems 100 allow the point-of-view or field-of-view of the system 100at which the image originates to be relayed to the other networkedsystems 100, or other wearable displays or computing devices, tofacilitate medical training, diagnosis, and treatment. As a result, thepoint of view or field of view of one system 100 can be presented toother networked systems 100 or computing systems.

In addition, the network of systems 100 may also be used to enable thevisualization of educational content, including but not limited tomedical training, surgical training and the like.

GPS

When the system 100 is configured with a GPS peripheral 350, the system100 is able to provide navigational information. As such, the system 100may be able to report the location of the device 100, communicate thelocation to another remote location over the communication network towhich the system 100 is connected. Furthermore, all navigationalinformation can be used by the system 100 to tag all data that isgathered by the system 100, such as images collected for example.

Microscope Imaging

The system 100 may also include microscopic imaging features. In oneaspect, the detector 120 may include the necessary optics to providemicroscopic imaging. In one aspect, the detector 120 may have built-inoptics to conduct microscopic imaging or may have interchangeableoptical components for microscopic imaging. In another aspect, themicroscope may be provided as a separate peripheral 350 that is coupledto the peripheral interface 300, such that the image supplied by themicroscope may be presented on the display or communicated through thenetwork other systems 100 and networked devices, as previouslydiscussed.

Medical Training

In one aspect, the memory unit of the system 100 may store software tosimulate a medical training procedure that is based on virtual realityor augmented reality. Two dimensional or three dimensional images orvideo may be stored at the memory unit of the system 100, or in a remoteserver coupled to the network to which the system 100 is connected,which enables visualization of educational content, such as medicaltraining and surgical training.

In another aspect, the training software may include audio-visualtraining tutorials with step-by-step instructions for carrying outparticular procedures via the display 110. In addition, the tutorialsmay outline tasks for how to prepare for an examination, how to operateultrasound, and how to position a patient. Ultrasound techniques, suchas how to manipulate the ultrasound probe and use the keyboard functionsof the ultrasound system may be included. The tutorials may also includevarious examination protocols; reference anatomy information withreference ultrasound images; procedures for how to make a diagnosis; andprocedures for how to treat patients and treatment tutorials may beincluded.

Alternatively, the system 100 may be worn by an instructor, such as ateaching surgeon, to provide teaching and instruction, such as theteaching of new surgical procedures and techniques. As such, the pointof view/field of view, as captured by the detector 120 of the system 100worn by the instructor is transmitted via the communication network 260to one or more students that are also wearing the system 100 forpresentation on their wearable display 110. In addition, theinstructor's point of view/field of view may be transmitted to any otherlocal or remote display, either wearable or non-wearable. In one aspect,a stereoscopic imaging and display system 100 enable capturing thestereoscopic view of the teacher surgeon, and transmit to other studentsalso wearing stereoscopic imaging and display systems 100, in real timeor near real time. The depth perception of stereoscopic images and videoprovide a more realistic training experience of medical procedures suchas surgeries. Furthermore, the students wearing stereoscopic imaging anddisplay systems 100 are able to see the training procedures conducted bythe teacher surgeon with the teacher's point of view and field of viewwith depth perception, in real time or near-real time, which is morerealistic than conventional method. Moreover, when the students areperforming similar procedures, the teacher wearing a stereoscopicimaging and display system 100 will be able to visualize thefield-of-view of one or more students. In one aspect, the teacher canmonitor students' performance by displaying different student'sstereoscopic view in a picture-in-picture format, or display severalwindows of different students concurrently. Thus, the pluralities ofstereoscopic imaging and display systems form a network for teachingmedical procedures and non-medical procedures, with a depth-perceptionand viewpoint-sharing.

Selective Bleed-Through

In one aspect, the light source 400 may have components that overlapwith emission spectra, referred to as bleed-through components. Thebleed-through components can be tunable to achieve desirable level ofbackground. For example, in the case of indocyanine green dye, if theemission filter is an 820 nm long-pass filter, the component ofillumination is >820 nm will pass through the emission filter (ifemission filter is 820 nm long pass filter) and become the background,or the bleed-through component. The illumination could have both 780 nmLEDs for fluorescence excitation and 830 nm LEDs for bleed-through. Bychanging the intensity of the 830 nm LEDs, the level of background canbe adjusted, which is useful in a variety of situations.

Medical or Surgical Guidance

Thus, with the components of the system 100 set forth, the particularmedical guidance functions enabled by the system will now be describedin detail. Thus, once the system has acquired the surgical navigationpre-operative images, the intra-operative images (e.g. fluorescenceimages), and the in-vivo, high-resolution imaging/sensing microscopyimages (e.g. fluorescence microscopy images), the previously discussedtracking and registration techniques are applied to the images. As aresult, all three image types are integrated, or superimposed,simultaneously together, so as to form a composite, co-registered image450 for presentation on the display 110 of the wearable system 100, asshown in FIG. 4. It should be appreciated that the detector 120 mayprovide stereoscopic imaging using the cameras 120A-B, as previouslydiscussed. This, allows a 3-dimensional image to be displayed forviewing by the surgeon via the wearable display 110, which simulatesnatural human vision, thereby allowing depth perception that is criticalin the guidance of medical assessments during surgeries.

The co-registered composite image 450 is configured, whereby thepre-operative image data, such as MRI (magnetic resonance imaging) datafor example, is segmented and processed into, and rendered, as a3-dimensional model having a plurality of 3D surfaces. It should beappreciated that any suitable segmentation process may be used,including: either automatic, manual or semi-automatic segmentationprocesses. In addition, segmentation can also be based on thresholdingmethods, clustering methods, compression-based methods, histogram-basedmethods, edge detection methods, region-growing methods, split-and-mergemethods, partial differential equation-based methods, parametricmethods, level set methods, fast marching methods, graph portioningmethods, watershed transformation methods, model based segmentationmethods, multi-scale segmentation methods, trainable segmentationmethods, and any combination thereof.

In one example, threshold based segmentation and manual segmentation maybe used prior to 3D rendering of the pre-operative images. In addition,one may use histogramming with a programmable threshold, Otsusegmentation for automatic thresholding, and a two-step drawing toolwhere regions are manually selected and then grouped using a clusteringalgorithm. Alternatively, manual region selection may be conducted. Inanother example, one may convert the rendered volume of the MRI imagesinto a surface mesh, and save the mesh as an object file. Histogramthresholding may be used to create inner and outer volumes of 3Drenderings, where the inner volume represents one organ, and the outervolume represents another organ.

Using tracking technologies, such as those previously discussed, the 3Dpre-operative model based on the MRI or other tomographic data is mappedto the physical dimensions of the patient 500, as shown in FIG. 4 who isundergoing the surgical procedure. That is, the mapping process resultsin the organs or other physical characteristic identified by thepre-operative 3D model of the patient to be correlated with thecorresponding organs of the patient 500 undergoing the surgicalprocedure. It should be appreciated that the pre-operative data may beacquired remotely by the system 100 via the communication network 260 orlocally by a portable memory unit configured to be coupled to or incommunication with the peripheral interface 300. In one aspect, thepre-operative image data may comprise any 3D volumetric data ortomographic image data. In addition, the pre-operative data used by thesystem 100 may comprise point cloud data, surface rendered data ormeshed data, MRI (magnetic resonance image) image data, computedtomography (CT) image data, positron emission tomography (PET) imagedata, single-photon emission computed tomography (SPECT), PET/CT,SPECT/CT, PET/MRI, gamma scintigraphy, X-ray radiography, ultrasound,and the like.

After the pre-operative image data is mapped to the patient 500,intra-operative imaging, such as fluorescence images, acquired by thedetector 120 in real-time or near real-time shows where a particularlesion or surgical site of interest is during the surgical procedure,and is added to the composite image 450. Thus, the intra-operativefluorescence image complements the 3D pre-operative MRI image that isdisplayed by the system 100. It should also be appreciated that theintra-operative fluorescence imaging also guides further assessment ofsmall lesions and other tissue structures using the in-vivo microscopyhand-held probe, as previously discussed. In particular, the operator orsurgeon wearing the system 100 uses the in-vivo microscopy hand-heldprobe to closely examine the diseased tissue or area, which is shown bythe intra-operative fluorescence imaging. The high-resolutionmicroscopic image from the hand-held microscopy probe is also shown onthe wearable display 110 in real-time as an inset or picture-in-pictureimage, as shown in FIG. 4. As such, FIG. 4 displays the ability of thesystem 100 to present via the display 110 pre-operative MRI-basedsurgical navigation imaging, intra-operative fluorescence imaging, andin-vivo microscopy imaging. This allows the user of the system 100 toconveniently identify the diseased tissue or lesion in the patient,while improving the diagnostic accuracy and decreasing the time neededfor assessment.

It should be appreciated that while the discussion of the system 100presented above enables the display 110 to show pre-operative images,intra-operative images, and in-vivo microscopy images together as acomposite image 450 each having various levels of transparency relativeto one another, any combination of one or more of the image types may bepresented on the display 110. For example, only pre-operative surgicalnavigation images and intra-operative fluorescence images may showntogether on the display 110; or only intra-operative images may be shownon the display 110.

It should also be appreciated that the system 100 may be configured,such that multiple intra-operative modalities are offeredsimultaneously, at the same time. For example, intra-operativefluorescence imaging and polarization imaging, as previously discussed,may be provided by the system 100 at the same time. Similarly,intra-operative fluorescence imaging and color imaging may be providedat the same time by the system 100.

In another aspect, the system 100 may be utilized to performgastrointestinal examinations, such as colonoscopies or esophagusexaminations. In such case, the surgical navigation pre-operative imageis based on a CT colonoscopy (virtual colonoscopy) image, theintra-operative imaging is acquired by an endoscope, and thehigh-resolution in-vivo microscopy sensing is achieved by anendomicroscopy probe. In addition, the system 100 may include a therapymodule 350 for attachment to the peripheral interface 300 that performsthe image-guided endoscopic surgery. As such, the system 100 allows theposition of the endoscope to be accurately tracked and displacedrelative to the 3D rendered models generated based on the CT colonoscopypre-operative image. Thus, surgical navigation can guide the assessmentof a lesion using the endoscope. Additionally, suspicious lesions can beassessed by the in-vivo endo-microscope probe to examine the microscopeand pathological details.

It should be appreciated that in addition to the types of displays 110previously discussed, the display 110 may also enable the side-by-sidedisplay of images, and may also include: anaglyph displays, polarized 3Ddisplays, active shutter 3D displays, interference filter 3D displaysystems, and the like. In addition, the display 110 may also comprisenon-stereoscopic display types as well. In addition, the display 110 maycomprise LCD (liquid crystal) microdisplays, LED (light emitting diode)microdisplays, organic LED (OLED) microdisplays, liquid crystal onsilicon (LCOS) microdisplays, retinal scanning displays, virtual retinaldisplays, optical see through displays, video see through displays,convertible video-optical see through displays, wearable projectiondisplays, and the like. In addition the display 110 may utilize aholographic display.

In addition, the detector 120 configurations previously discussed withregard to FIGS. 3A-E may be used to provide the intra-operative images,such as the fluorescence images, used for generating the composite image450 for presentation on the display 110 by the system 100. For example,the cameras 120A-B may be configured to provide stereoscopicfluorescence imaging.

In addition, the light source 400 is used during intra-operativeimaging, such as fluorescence imaging, and may include but is notlimited to: a non-coherent light source, such as a Xenon lamp, a halogenlamp and LEDs. In addition, coherent light sources may be used, such aslasers, and laser diodes. Furthermore, various fluorescence tracers maybe used to initiate fluorescence at the surgical site or tissue ofinterest 130 may be used by the system 100 for use in different lightsource 400 spectra, including the UV (ultra-violet) range, visiblerange, and infrared (IR) range may be used with the system 100. In oneaspect, near-infrared (NIR) fluorescence imaging may be performed usingindocyanine green as a fluorescence tracer.

In addition to intra-operative fluorescence imaging, other types ofintra-operative imaging may be performed, including: polarizationimaging, absorption imaging, oxygen saturation information imaging, orany combination thereof. Furthermore, intra-operative imaging may beperformed for any suitable surgical procedure, such as open surgery,endoscopic surgery, laparoscopic surgery, or any combination thereof.

It should also be appreciated that the intra-operative imaging probeused to acquire the intra-operative images may comprise an ultrasoundprobe, an endoscope, a laparoscope, a bronchoscope and the like.

It should also be appreciated that in-vivo microscopic imaging may beperformed by any suitable in-vivo microscopy probe, such as an in-vivofluorescence/reflectance microscopy probe. In addition, various in-vivomicroscopy detection and techniques may be used by the system 100,including color microscopy, reflectance microscopy, fluorescencemicroscopy, oxygen-saturation microscopy, polarization microscopy,infrared microscopy, interference microscopy, phase contrast microscopy,differential interference contrast microscopy, hyperspectral microscopy,total internal reflection fluorescence microscopy, confocal microscopy,non-linear microscopy, 2-photon microscopy, second-harmonic generationmicroscopy, super-resolution microscopy, photoacoustic microscopy,structured light microscopy, 4Pi microscopy, stimulated emissiondepletion microscopy, stochastic optical reconstruction microscopy,ultrasound microscopy, and any combination thereof.

In addition, the in-vivo probe may comprise a handheld\that has not yetreached the microscopic resolution. Non-microscopic imaging methods,which may be used by the in-vivo probe 350 may include reflectanceimaging, fluorescence imaging, Cerenkov imaging, polarization imaging,ultrasound imaging, radiometric imaging, oxygen saturation imaging,optical coherence tomography imaging, infrared imaging, thermal imaging,photoacoustic imaging, spectroscopic imaging, hyperspectral fluoroscopyimaging, gamma imaging, x-ray computed tomography imaging, or anycombination thereof. It should also be appreciated that the in-vivomicroscopy probe may comprise an endoscope, a laparoscope, abronchoscope, an angioscope, a catheter for angiography.

In another aspect, the in-vivo probe may comprise a non-imaging device,such as a sensing device, including a handheld spectrophotometer orfiber-based spectrometers. Using the in-vivo sensing probe, variousspectroscopies may be realized, such as various optical spectroscopies,absorption spectroscopy, fluorescence spectroscopy, Raman spectroscopy,coherent anti-Stokes Raman spectroscopy (CARS), surface-enhanced Ramanspectroscopy, Fourier transform spectroscopy, Fourier transform infraredspectroscopy (FTIR), multiplex or frequency-modulated spectroscopy,x-ray spectroscopy, attenuated total reflectance spectroscopy, electronparamagnetic spectroscopy, electron spectroscopy, gamma-rayspectroscopy, acoustic resonance spectroscopy, Auger spectroscopy,cavity ring down spectroscopy, circular dichroism spectroscopy, coldvapour atomic fluorescensce spectroscopy, correlation spectroscopy,deep-level transient spectroscopy, dual polarization interferometry, EPRspectroscopy, force spectroscopy, Hadron spectroscopy, Baryonspectroscopy, meson spectroscopy, inelastic electron tunnelingspectroscopy (JETS), laser-induced breakdown spectroscopy (LIBS), massspectroscopy, Mossbauer spectroscopy, neutron spin echo spectroscopy,photoacoustic spectroscopy, photoemission spectroscopy, photothermalspectroscopy, pump-probe spectroscopy, Raman optical activityspectroscopy, saturated spectroscopy, scanning tunneling spectroscopy,spectrophotometry, time-resolved spectroscopy, time-stretchspectroscopy, thermal infrared spectroscopy, ultraviolet photoelectronspectroscopy (UPS), video spectroscopy, vibrational circular dichroismspectroscopy, x-ray photoelectron spectroscopy (XPS), or any combinationthereof.

Based on the foregoing, the advantages of the present invention arereadily apparent. The main advantage of this system to provide aplurality of ways to guide medical procedures, such as surgicalprocedures, through surgical navigation (e.g. pre-operative imaging),intraoperative imaging and high-resolution in-vivo imaging/sensing (e.g.microscopy) using a single wearable system. Still another advantage ofthe present invention is that medical information at all scales isavailable at the same time for viewing by a surgeon, whereby surgicalnavigation provides whole-body information based on MRI or CT imaging;intraoperative imaging offers real-time information for the organ ofinterest; and microscopy offers microscale information, which is moreconvenient and faster than conventional pathology reporting techniques.Another advantage of the present invention is that the wearable systemis easy to wear and use, and is user-friendly to operate. Yet anotheradvantage of the present invention is that the depth perception (i.e.stereoscopic vision) offered by the system is beneficial in guidingsurgery, which surpasses the performance of planar imaging and 2Dmonitor display. Another advantage of the present invention is that thewearable device provides a way to correlate surgical navigation,intraoperative imaging and high-resolution imaging together, viaaccurate tracking and image registration techniques. Yet anotheradvantage of the present invention is that the system provides wirelesscommunication, which allows the wearable device to communicate withmultiple communication devices and wearable devices remotely andlocally; whereby the remote user of the system has the option to viewstereoscopic information and talk to local clinicians in near real-time.Another advantage of the present invention is that the wearable deviceis self-contained, which facilitates its use in rural areas, developingcountries, and in first responder and security/defense applications.Still another advantage of the present invention is that the wearabledevice along with other remotely located wearable devices may be usedfor medical training where the actual medical information viewed by thesurgeon can be simultaneously viewed by students. Another advantage ofthe present invention is that the wearable device may be low cost; maybe used in any diverse surgical settings, such as veterinary medicine.Another advantage of the present invention is that a platform technologyis provided by the system, which can be used for nearly all medicalinterventions where radiographic findings are important.

Thus, it can be seen that the objects of the present invention have beensatisfied by the structure and its method for use presented above. Whilein accordance with the Patent Statutes, only the best mode and preferredembodiment has been presented and described in detail, it is to beunderstood that the present invention is not limited thereto or thereby.Accordingly, for an appreciation of the true scope and breadth of theinvention, reference should be made to the following claims.

What is claimed is:
 1. An imaging and display system for guiding medicalinterventions comprising: a display adapted to be worn by a user; adetector coupled to said display, said detector configured to captureintra-operative images from a target; and a computing unit coupled tosaid display and to said detector, said computing unit adapted to storepre-operative images.
 2. The imaging and display system of claim 1,wherein said display presents said pre-operative image and saidintra-operative image simultaneously.
 3. The imaging and display systemof claim 1, wherein said display presents said pre-operative image andsaid intra-operative image simultaneously as a composite, co-registeredimage on said display.
 4. The imaging and display system of claim 1,further comprising: a communication interface coupled to said computingunit to enable communication with at least one other display.
 5. Theimaging and display system of claim 1, further comprising: a peripheralinterface coupled to said computing unit, said peripheral interfaceadapted to communicate with one or more peripherals.
 6. The imaging anddisplay system of claim 1, further comprising: a peripheral interfacecoupled to said computing device, said peripheral interface adapted tocommunicate with one or more peripherals, wherein said peripheralcomprises a microscope (in vivo, hand-held or conventional) selectedfrom the group consisting of: a fiber microscope, a handheld microscope,a color microscope, a reflectance microscope, a fluorescence microscope,an oxygen-saturation microscope, a polarization microscope, an infraredmicroscope, an interference microscope, a phase contrast microscope, adifferential interference contrast microscope, a hyperspectralmicroscope, a total internal reflection fluorescence microscope, aconfocal microscope, a non-linear microscope, a 2-photon microscope, asecond-harmonic generation microscope, a super-resolution microscope, aphotoacoustic microscope, a structured light microscope, a 4Pimicroscope, a stimulated emission depletion microscope, a stochasticoptical reconstruction microscope, an ultrasound microscope, andcombinations thereof.
 7. The imaging and display system of claim 1,further comprising: a peripheral interface coupled to said computingdevice, said peripheral interface adapted to communicate with one ormore peripherals, wherein said one or more peripherals comprises aimaging system selected from the group consisting of: an ultrasoundimager, a reflectance imager, a diffuse reflectance imager, afluorescence imager, a Cerenkov imager, a polarization imager, aradiometric imager, an oxygen saturation imager, an optical coherencetomography imager, an infrared imager, a thermal imager, a photoacousticimager, a spectroscopic imager, a Raman spectroscopic imager, ahyper-spectral imager, a fluoroscope imager, a gamma imager, an X-raycomputed tomography imager, an endoscope imager, a laparoscope imager, abronchoscope imager, an angioscope imager, and an imaging catheterimager.
 8. The imaging and display system of claim 1, furthercomprising: a peripheral interface coupled to said computing device,said peripheral interface adapted to communicate with one or moreperipherals, wherein said peripheral comprises a spectrometer selectedfrom the group consisting of: an optical spectrometer, an absorptionspectrometer, a fluorescence spectrometer, a Raman spectrometer, acoherent anti-stokes Raman spectrometer, a surface-enhanced Ramanspectrometer, a Fourier transform spectrometer, a Fourier transforminfrared spectrometer (FTIR), a diffuse reflectance spectrometer, amultiplex or frequency-modulated spectrometer, an X-ray spectrometer, anattenuated total reflectance spectrometer, an electron paramagneticspectrometer, an electron spectrometer, a gamma-ray spectrometer, anacoustic resonance spectrometer, an auger spectrometer, a cavity ringdown auger spectrometer, a circular dichroism auger spectrometer, a coldvapour atomic fluorescence auger spectrometer, a correlationspectrometer, a deep-level transient spectrometer, a dual polarizationinterferometry, an EPR spectrometer, a force spectrometer, a Hadronspectrometer, a Baryon spectrometer, a meson spectrometer, an nelasticelectron tunneling spectrometer (IETS), a laser-induced breakdownspectrometer (LIBS), a mass spectrometer, a Mossbauer spectrometer, aneutron spin echo spectrometer, a photoacoustic spectrometer, aphotoemission spectrometer, a photothermal spectrometer, a pump-probespectrometer, a Raman optical activity spectrometer, a saturatedspectrometer, a scanning tunneling spectrometer, a spectrophotometryspectrometer, time-resolved spectrometer, a time-stretch spectrometer, athermal infrared spectrometer, an ultraviolet photoelectron spectrometer(UPS), a video spectrometer, a vibrational circular dichroismspectrometer, and an X-ray photoelectron spectrometer (XPS).
 9. Theimaging and display system of claim 1, further comprising: a peripheralinterface coupled to said computing device, said peripheral interfaceadapted to communicate with one or more peripherals, wherein saidperipheral comprises a tracking system selected from the groupconsisting of: an optical tracking system, an electromagnetic trackingsystem, a radio frequency tracking system, a gyroscope tracking system,a video tracking system, an acoustic tracking system, and a mechanicaltracking system.
 10. The imaging and display system of claim 9, whereinthe movement of said detector is configured to be tracked by saidtracking system, such that the position of said intra-operative imagecaptured by said detector is adjusted to maintain registration with saidpre-operative image.
 11. The imaging and display system of claim 5,wherein said one or more peripherals comprises a tracking system and animaging or sensing probe, said probe capturing imaging or sensing datafor composite presentation with said intra-operative image and saidpre-operative image on said display.
 12. The imaging and display systemof claim 11, wherein said probe comprises an in-vivo microscopy probe.13. The imaging and display system of claim 11, wherein the movement ofsaid in-vivo microscopy probe is configured to be tracked by saidtracking system, such that the position of said probe is presented onsaid display.
 14. The imaging and display system of claim 1, whereinsaid display comprises a stereoscopic display.
 15. The imaging anddisplay system of claim 1, wherein said detector comprises astereoscopic detector.
 16. The imaging and display system of claim 1,wherein said display presents a plurality of different imaging orsensing data in a picture-in-picture format.
 17. The imaging and displaysystem of claim 1, wherein said detector is configured to detect one ormore types of said intra-operative images selected from the groupconsisting of: a fluorescence image, a reflectance image, a color image,a light absorption image, a light scattering image, an oxygenationsaturation image, a polarization image, a thermal image, an infraredimage, a hyperspectral image, a light field image, a fluorescencelifetime image, a bioluminescence image, a Cerenkov image, aphosphorescence hyperspectral image, a spectroscopic image, achemilluminescence image and a scintillation image.
 18. The imaging anddisplay system of claim 1, wherein said pre-operative images comprisetomographic images.
 19. The imaging and display system of claim 1,wherein said pre-operative images comprise 3D models processed frompre-operative tomographic data.
 20. The imaging and display system ofclaim 1, wherein said computing unit is configured to perform the stepscomprising: computing a transformation matrices between a pre-operativeimage space, an intra-operative object/patient space and anintra-operative image space; and co-registering said pre-operative imagespace, said intra-operative image space and said intra-operativeobject/patient space.
 21. The imaging and display system of claim 1,wherein said computing unit is configured to perform the stepscomprising: computing a transformation matrices between a pre-operativeimage space, an intra-operative object/patient space, an intra-operativeimage space and a peripheral image space; and co-registering saidpre-operative image spaces, said intra-operative image space, saidperipheral image space, and said intra-operative object/patient space.22. The imaging and display system of claim 1, further comprising: alight source for illuminating said target.
 23. The imaging and displaysystem of claim 22, wherein said light source comprises one or morewhite light-emitting diodes and one or more band-rejection opticalfilters, wherein the frequencies of light emitted by said light sourcethat overlaps with a fluorescence emission from said target is blockedby said band-rejection optical filters.
 24. The imaging and displaysystem of claim 1, further comprising: a light source for illuminatingsaid target, wherein said light source comprises one or more projectorsand one or more spectral filters.
 25. The imaging and display system ofclaim 1, further comprising: a light source wherein said light sourcecomprise a pulsed illumination device, or may utilize frequencymodulation or pulse-duration modulation.
 26. The imaging and displaysystem of claim 1, further comprising: a light source, wherein saidlight source emits an illumination beam that is provides an adjustablelevel of light frequencies that overlap with an emission spectra of saidtarget.
 27. The imaging and display system of claim 1, furthercomprising: a peripheral interface coupled to said computing unit, saidperipheral interface adapted to communicate with one or moreperipherals, wherein said peripherals comprise one or more trackingsystems, wherein said tracking systems comprise LEDs and spectralfilters.
 28. The imaging and display system of claim 1, furthercomprising: a peripheral interface coupled to said computing device,said peripheral interface adapted to communicate with one or moreperipherals, wherein said peripherals comprise one or more trackingsystems, wherein said tracking systems comprise software that enabletopology sampling using a tracked handheld imaging probe or a trackedhandheld sensing probe.
 29. The imaging and display system of claim 1,wherein said computing unit stores educational or medical trainingcontents.
 30. A imaging and display system comprising: a plurality ofgoggles, each including: a stereoscopic display for viewing by the eyesof one wearing the goggle, a stereoscopic detector coupled to saidstereoscopic display, said detector having a field of view andprojecting an image within that field of view onto said display, and acommunication interface linking each of said plurality of goggles tocommunicate with each other.